This PR is intended to remove the global out/ folder lock in favor of finer grained per-task or per-bootstrap-phase read/write locks, with handling for the various "global" out/ folder files (e.g. mill-chrome-profile.json,
mill-profile.json, mill-console-tail, etc.) to preserve their usefulness in the presence of concurrent runs.

Locking

• During task execution, we take a read lock for each task, if it needs to be re-evaluated we then take a write lock, and when evaluation is done we keep the read lock to ensure it remains stable for use by downstream tasks until all tasks downstream have completed, at which point the read lock is released.

  • The DAG nature of the build graph means that we do not need to worry about deadlocks caused by inter-task read/write locking: blocking is either caused by a write lock waiting on a read lock (in which case the read lock will eventually get released when the downstream code finishes. The downstream code could also get blocked, but that block will similarly eventually terminate for the same reason) or a read lock waiting on a write lock (in which case the write lock will eventually downgrade to read when that task finishes executing)

• When evaluating meta-builds, we first take a read lock and then run selective execution as a non-task-executing way of checking if nothing changed. Only if selective execution detects changes do we take the write lock and evaluate that meta-level's tasks, and convert to a read lock after it is done so it remains stable while we use it.

• This change is meant to only affect daemon mode, since we don't have reliable filesystem locking we can rely on for non-daemon mode (e.g. docker folder mounts don't always support locks properly). mill-out-lock is turned into a process-level lock, such that either a single MillDaemonMain process or MillNoDaemonMain process can claim it. If a MillDaemonMain claims it then it can support multiple concurrent runs with fine-grained locking internally, but if a MillNoDaemonMain claims it then only that one run can take effect

Splitting Of Global State and Per-Launcher State

• The various out/mill-* files are turned into symlinks to out/mill-run-*/* files: the out/mill-* file pointing at the most recent out/mill-run-*/* file to be created, but the older out/mill-run-*/* files kept around in case people need them (e.g. due to concurrent use)

• The old RunnerState was split into a daemon-wide RunnerSharedState and a per-launcher RunnerLaunchState. This lets us mostly re-use the immutable RunnerSharedState between launchers, while keeping the per-launcher state separate. RunnerSharedState only gets updated under a meta-build write lock

• Task.Workers are shared between concurrent launchers; this is necessary for efficiency, to avoid e.g. spawning duplicate JvmWorkerMain or ZincWorker classloaders that can be very expensive.

• MillMain0.main0 no longer takes and returns the RunnerState at the end of each execution, since that only works in a single-threaded environment. Instead it receives a sharedState as an AtomicReference[RunnerSharedState] that it can update at any time as the necessary state gets computed

BSP Changes

• With this PR, we are swapping the default back for BSP to share the same out/ folder as normal execution, as the fine grained locking should keep things from blocking or corrupting each other

• Replaced the long-lived BSP evaluator cache and iteration loop with a BspBootstrapBridge such that each request bootstraps fresh evaluators per-call. This makes BSP requests behave like CLI commands, importantly behaving the same w.r.t. locking and concurrency (e.g. meta-build lock, task locks, etc.). We make some tweaks to the meta-build bootstrap process (e.g. using selective execution as a pre-filter) to try and mitigate the slowdown of going through the bootstrap process every BSP request where we did not before

• BSP requests are now handled in a multithreaded fashion, sharing the same locking/concurrency model as concurrent CLI commands. This keeps BSP and CLI in sync, since this PR already adds concurrency support for the CLI, we also add it for BSP

• The caching of BspEvaluators when the meta-build is broken is removed. It is hard to implement with the new architecture in the presence of concurrency, and anyway partial meta-build failures still generate partial BSP output and so e.g. editing the build.mill files while they're broken should still work

Misc

• When a launcher detects a version mismatch, it now lets the old daemon finish its in-flight commands before restarting it, instead of force-killing and stranding peer launchers with EOF. The launcher just deletes processId and waits for daemonLock to free; the daemon's processId-watcher flips into drain mode (rejecting new connections, waiting indefinitely for active ones to complete with their normal results), so peers run to completion as if nothing happened.